fix: Place only READY SubConns on the hashring so RPCs are not queued behind a connecting or failed backend - #4

Disclaimer: written by AI 🙃

Problem

The picker returns whichever SubConn the request key hashes to, without checking its connectivity state. If that backend is CONNECTING or in TRANSIENT_FAILURE, gRPC parks the RPC until the SubConn becomes READY or the connect attempt times out (20s by default). The ring only changes when the resolver adds or removes addresses, so until that happens every request hashed to a dead backend hangs.

How it was observed

Killing one SpiceDB node in a two-node cluster (SIGINT or SIGTERM) made the surviving node stop answering any request that dispatched to the dead peer. Requests with a 10s deadline failed with DeadlineExceeded at exactly +10s and +20s; once the dial timed out, everything failed fast with Unavailable. The killed node itself drained cleanly. In Kubernetes this shows up as a burst of DeadlineExceeded and Unavailable on the remaining replicas during every rolling deploy, lasting until the endpoint list drops the pod IP.

Fix

Ring membership now follows SubConn state: a backend is placed on the ring when its connection becomes READY and removed when it leaves READY. Keys whose closest backend is down are served by the next closest ready backend, which is safe because any node can serve any request. When nothing is ready, RPCs queue until a backend comes up, or fail fast once the balancer reports TRANSIENT_FAILURE.

Two new tests cover this. A unit test flips one backend to TRANSIENT_FAILURE and checks the pick moves to the ready one. An end-to-end test runs a real gRPC server next to a TCP listener that never completes the HTTP/2 handshake and asserts an RPC hashed to it is answered promptly instead of hitting its deadline. Both failed before the change.